A number of video coding standards support variable frame rates, e.g., H.263 and MPEG-4. With variable frame-rates, any number of frames, or objects in the case of MPEG-4, can be skipped during the coding of the output video. That is, the skipped frames remain uncoded. With these video coding standards, the encoder may choose to skip frames of a video to either satisfy buffer constraints, or to optimize the video coding process. However, most encoders only skip frames to satisfy buffer constraints. Buffer constraints are usually due to bit-rate (bandwidth) limitations. The coder is forced to skip frames when insufficient bandwidth causes the buffer to fill up. Consequently, it is not possible to add any additional frames to the buffer, and these frames remain uncoded (skipped) until there is room in the buffer to store a new coded frame. This type of frame skipping can degrade the quality of the video because the content of the video is not considered. Note that skipping frames effectively reduces the frame-rate.
It is a problem to provide an optimal strategy for coding a video. Specifically, the video could be coded at a higher frame-rate having a lower spatial quality, or a lower frame-rate having a higher spatial quality. This trade-off between spatial and temporal quality is not a simple binary decision, but rather a decision over a finite set of coding parameters (constraints). Obviously, the best set of coding parameters will yield the optimal rate-distortion (R-D) curve that maximizes the frame-rate while minimizes the distortion. The two parameters of interest are the number of frames per second (fps or frame-rate) and a quantizer (Q) parameter. A higher quantizer parameter increases the spatial distortion. Lowering the frame rate, by skipping frames, reduces both the spatial and temporal distortion. In the known prior art, the distortion is measured only for coded frames, and is expressed as the mean-squared error (MSE) between pixels in the original video and the compressed video. That is, the prior art methods have two problems, only spatial distortion in coded frames is considered, and uncoded frames contributing to both the spatial and temporal distortion are not considered at all.
Generally, prior art optimized coding methods do not consider the temporal aspect of rate-distortion, see H. Sun, W. Kwok, M. Chien, and C. H. John Ju, “MPEG coding performance improvement by jointly optimizing coding mode decision and rate control,” IEEE Trans. Circuits Syst. Video Technol., June 1997, T. Weigand, M. Lightstone, D. Mukherjee, T. G. Campbell, S. K. Mitra, “R-D optimized mode selection for very low bit-rate video coding and the emerging H.263 standard,” IEEE Trans. Circuits Syst. Video Technol., and April 1996, J. Lee and B. W. Dickenson, “Rate-distortion optimized frame type selection for MPEG encoding,” IEEE Trans. Circuits Syst. Video Technol., June 1997. Generally, it is assumed that the frame-rate is fixed.
These methods consider optimizations on the quantizer parameter, H. Sun, W. Kwok, M. Chien, and C. H. John Ju, “MPEG coding performance improvement by jointly optimizing coding mode decision and rate control,” IEEE Trans. Circuits Syst. Video Technol., June 1997, mode decisions for motion and block coding, T. Weigand, M. Lightstone, D. Mukherjee, T. G. Campbell, S. K. Mitra, “R-D optimized mode selection for very low bit-rate video coding and the emerging H.263 standard,” IEEE Trans. Circuits Syst. Video Technol., April 1996, and frame-type selection, J. Lee and B. W. Dickenson, “Rate-distortion optimized frame type selection for MPEG encoding,” IEEE Trans. Circuits Syst. Video Technol., June 1997. Such methods can achieve an optimum coding when the frame-rate is fixed, and the bit-rate can be met for the given frame-rate. However, these methods are less than optimal for varying frame-rates.
It should be noted that the trade-off between spatial and temporal quality, while coding, has been described by F. C. Martins, W. Ding, and E. Feig, in “Joint control of spatial quantization and temporal sampling for very low bit-rate video,” Proc. ICASSP, May 1996. However, in their method, the trade-off was achieved manually.
Therefore, it is desired to provide a method and system for encoding a video subject to a variable frame-rate, while minimizing the total average distortion.